CI Test Results

Run: #29252241893 | Commit: 9b7fb5f | Duration: 15m 57s (longest job)

✅ All 32 test jobs passed

Status Overview

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0

Updated: 2026-07-13 13:38:05 UTC

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/java-profiler | reliability-chaos-amd64: [profiler, gmalloc, 25.0.3-tem]     

Useful? React with 👍 / 👎

Reliability & Chaos Results

❌ 1 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/124258703

Chaos harness crashed (RC=124)

Review

Scope: 3 files, +739/-25 — README.md, utils/README.md, utils/run-containers-tests.sh. No CI workflow files touched; this only extends the local dev/test script that mirrors CI's container test matrix (glibc/musl × JDK × arch × config), adding --matrix, --run, --fail-fast, and multi-value (all/comma-list) dimension filters, plus a status table and markdown/JSON summary reports.

Overview

The script gains dimension expansion (expand_dimension, expand_jdk_dimension), a skip_reason() table that mirrors known CI exclusions (musl+j9/graal, graal+asan, glibc/aarch64 8-11, musl+sanitizers), a matrix preview/confirmation flow, sequential matrix execution with --fail-fast, and report writers. Docs in both READMEs are updated consistently with the new flags.

Bug found (verified by running the script)

utils/run-containers-tests.sh:907:

if (( MATRIX_TOTAL > 1 || MATRIX_RUNNABLE > 1 || MATRIX_MODE )); then

MATRIX_MODE holds the string "true"/"false", not 1/0. In a bash arithmetic context, a variable holding a non-numeric string is itself re-evaluated as a variable name — since there's no shell variable named true, it silently resolves to 0. So MATRIX_MODE never actually contributes to this condition.

Confirmed with:

./run-containers-tests.sh --matrix --libc=glibc --jdk=21 --arch=x64 --config=debug --container=docker
# → "Error: container runtime 'docker' not found" — i.e. it fell through to the single-cell
#   execution path instead of showing the matrix preview/confirmation, even with --run.

Impact is narrow — bare --matrix or partial filters expand to >1 cell and work correctly (verified separately). But it's a real correctness bug in exactly the "preview a specific single-cell matrix explicitly" path the flag is meant to support, and "boolean flag in arithmetic context" is a bug class worth fixing before it recurs elsewhere. Fix:

if (( MATRIX_TOTAL > 1 || MATRIX_RUNNABLE > 1 )) || $MATRIX_MODE; then

Minor / non-blocking

• matrix_command() (run-containers-tests.sh:336) is defined but never called anywhere in the script — dead code, presumably superseded by the inline command-building in run_matrix(). Worth deleting.
• Heavy use of eval for indirect array access (append_unique_line, read_lines_into_array, read_string_into_array) works but is easy to get wrong later; a bash-4+ local -n nameref would be safer if the minimum bash version allows it — just flagging the tradeoff (macOS ships bash 3.2 by default, where namerefs aren't available, so eval may be the pragmatic choice here).
• No automated test coverage for the new flag-parsing/expansion logic (understandable for a bash dev-utility script, but a quick self-check invocation, e.g. a --matrix dry run, in CI would catch regressions like the one above).

Verdict

Nothing here is a security or CI-breaking risk — it's an opt-in local dev tool with no production/CI wiring changed. The MATRIX_MODE arithmetic bug is real and demonstrable but narrow in scope. Flagging as should-fix-before-merge rather than a hard blocker — it's a quick one-line fix, and the documented usage patterns all work correctly as tested.

@rkennke thanks for the review; I've applied your suggestions in 6462618.

Benchmark Results (commit afa13dd)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124061231 Commit: afa13dd56dcae17b00897bf5c3968cf5354f0303

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Benchmark Results (commit 9efcd6d)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124258722 Commit: 9efcd6dff4415f512ecd19baf58fa2cfbe860830

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Benchmark Results (commit afe7daf)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124304145 Commit: afe7daf9836131c3ccc12238c9d4fbb66fc15934

⚠️ Significant outliers

• 🔴 future-genetic (JDK 25): runtime +6.1% (2008→2130 ms)
• 🔴 reactors (JDK 21): runtime +7.8% (15499→16710 ms)